Archivematica outside the box: Piloting a common approach to digital preservation at the Five College Libraries

Purpose ​– This work aims to provide a report on adopting a consortial model of collaboration toward understanding digital preservation practice. Design/methodology/approach​ – This work provides a case study detailing the work and outcomes of a digital preservation pilot project undertaken by the Five College Libraries between 2014 and 2015. Findings ​– Digital preservation is a broad endeavor and rapidly developing facet of digital collections and institutional repositories; yet, it is often an area that is not fully understood or implemented by many libraries and archives, largely because institutions lack the necessary resources to do it alone. Working across institutional lines provides a possible solution to overcoming resource limitations and general challenges for pursuing robust digital preservation programs. Research limitations/implications​ – Findings reported in this work are based on a limited-scope pilot project. Several questions laid out during the pilot remain unanswered at its close. Originality/value ​– This paper provides insight into an experimental process rarely reported in library and information science literature. The goal of the paper is to provide a reference point for institutions pursuing a consortial approach to the challenges of applied digital preservation practice. Introduction Between July 2014 and May 2015, library staff from the Five Colleges (Amherst, Hampshire, Mount Holyoke, Smith and UMass-Amherst) undertook a project to explore facets of digital accessioning and preservation using an open-source platform called Archivematica (2016). This pilot project was spearheaded by a working group assembled by the Five College Digital Preservation Task Force, a consortial group focused on the challenges of digital preservation with interest in pursuing a collaborative approach to digital stewardship at the Five Colleges (FCs). The pilot emphasized this interest in consortial collaboration by customizing Archivematica’s framework around centralized archival storage of digital objects and by investigating shared workflows for digital object creation across the colleges. The pilot met with several challenges along the way, but succeeded most significantly in providing a venue for shared conversations and problem-solving around common digital preservation pitfalls. In the end, the working group was able to gain experience with applied digital preservation activities and set forth recommendations toward further collaboration in the FCs. The Five College Digital Preservation Task Force Concept and Mission Finding the necessary resources to establish a digital preservation program and implement a digital preservation repository is a challenge for many institutions due to budget and staffing limitations. Realizing this, the libraries in the FC Consortium (University of Massachusetts Amherst, Amherst College, Hampshire College, Mount Holyoke College, and Smith College) decided to explore the possibility of working together in establishing an ongoing, sustainable program for digital preservation across the colleges. Existing Five College consortial efforts, such as the management of a shared integrated library system (ILS) and a shared print depository, provided models of successful Five College collaboration and influenced discussions of a shared approach to digital preservation across the libraries. In February of 2011, a Five College Digital Preservation Task Force was formed to discuss the advantages and disadvantages of collaboration and to research how the Five Colleges might work together on long-term preservation of their digital content. After completing an environmental scan of each institution and interviewing representatives from other digital preservation collaboratives, the group became convinced that collaborating to achieve sustainable, long-term preservation was an obvious goal for the Five Colleges. There were many compelling reasons why they thought cooperation would help with achieving the goal of a sustainable digital preservation program. Perhaps the biggest incentive to working together was cost efficiency. One potential model of collaboration discussed would allow the Five Colleges to develop a single technological infrastructure with a single storage system and a single point of management. Most forms of collaboration under consideration would at least allow the Five Colleges to benefit from pooled knowledge. Expertise in particular areas, such as electronic records ingest, metadata management, and repository systems had been developed to differing extents in the five institutions. Working together would allow each of the Five Colleges to benefit from local experience that was either isolated or lacking at their own institution. Leadership at the Five College Libraries agreed that it was the right time to collaborate in this area. Each of the Five Colleges were already either creating and/or taking responsibility for born-digital objects, and all were aware of the need to preserve their digital content in general, but individually they were moving slowly, in limited fashion, or not at all toward developing a robust program for digital preservation. It seemed likely that few of the institutions within the Five Colleges could develop a fully realized, sustainable digital preservation program on their own. The risks of not collaborating were great. At best, each of the Colleges would spend money on multiple, redundant positions and systems, all to achieve the same goal. At worst, some or all of the institutions would not be able to support digital preservation at all, and would lose essential digital content – digital content that they expended resources to create; that may be necessary for legal purposes; that would create a gap in the historical, cultural, and scholarly record. In 2012 the Task Force submitted a report to the Five College Librarians Council that outlined the following recommendations: ● Each institution should make an expressed commitment to preserve its digital content, documented through digital preservation policies. ● Each institution should conduct a review of current practices for creating digital objects, and develop shared guidelines to ensure that they are creating preservation-ready digital objects. ● Wherever possible the FC Libraries should collaborate and investigate methods toward the implementation of a trusted digital preservation repository. By 2014, members of the Task Force had invested significant effort into research and assessment of their institution’s readiness to undertake a digital preservation program. The group had also provided professional development opportunities for the community to further engage with decision-making around digital preservation in their work. Examples included hosting Cornell and MIT Libraries’ Digital Preservation Management workshop (dpworkshop, 2016) and the Digital POWRR (Preserving Digital Objects with Restricted Resources) workshop (digitalpowrr, 2016), as well as preparing a digital preservation readiness guide and checklist (Five Colleges Inc., 2016). After facilitating these educational opportunities in the FC community and opening up discussions around standard practices for dealing with digital content over the long term, the Task Force determined that it was time to apply what it had learned in a practical, hands-on way. Focusing complex decision-making and experimentation around a particular toolset appeared to be an effective tactic, and after some investigation of digital preservation services, the Task Force agreed to engage in a pilot project and to install and test Archivematica. Courtney Mumma, the former Archivematica and AtoM Community Development Consultant for Artefactual Systems, described Archivematica as: [. . .] a suite of many open source tools knit together and combined with other functionality via micro-services to achieve OAIS-compliant preservation actions on your digital content. Users have many configuration options and opportunities to make decisions about their content as it goes through Ingest[1]. Archivematica is a web application suite comprised of two major components a Storage Service for managing long-term disposition of digital assets in the form of archival packages, and a Client panel (or Pipeline) for processing digital accessions, i.e. transforming sets of files into submission packages, before they are ingested by the Storage Service[2]. There are numerous ways that these two components can be configured to interact, and the setup can vary depending on whether it is used by a single department within an organization, cross-departmentally, or by a consortium of organizations. Generally, an installation of Archivematica includes a single Storage Service server and one or more Client machines. It is possible to install both the Client and the Storage Service on the same server, but for larger departmental or institutional use it also makes sense to install Clients on multiple local machines, i.e. personal laptops or desktop computers, that have network access to a single remote Storage Service server. However it is important to note that at the time the Task Force began investigating Archivematica there was little or no literature or open guidelines that discussed configuring the system for consortial use.[3]. It became clear to the Task Force that, even with the experimental and under-documented nature of their undertaking, the limited budget and resources available to the pilot would not allow for hiring outside developers or professional consultants to help build their test system. Instead they invested any available local resources to understanding the potential of Archivematica as a tool for grappling with digital preservation policies and decision-making, and attempted to experiment and provide novel solutions to obstacles that arose during the course of the pilot. Overview of the Archivematica Pilot The goal of the six month pilot was to give each institution in the Five Colleges an opportunity to wrestle with the practicalities of digital object creation and management and give the consortium a chance to outline the challenges and opportunities of shared policy and infrastructure development for digital preservation. Though the project was focused around a particular software solution, it was equally intended to evaluate each institution's capabilities and readiness for digital stewardship. The Task Force intended for this pilot to provide opportunities for all member institutions to ask questions on their campuses related to everything from high-level policy development to server space and resource allocation for digital preservation. In addition to the goals and outcomes outlined for the pilot project on a consortial level, each institution had individual goals for the project, and each institution decided which area of focus it would take while testing Archivematica. In order to test a wide range of formats and workflows within Archivematica in a limited amount of time, each of the Five Colleges planned to investigate a particular format or workflow and document their process, policy decisions, experience, and the results of their testing: ● Amherst College planned to test Archivematica as a tool for ingesting digitized images and textual materials. Specifically, they planned to ingest tiff and jpeg images of some of their unique and rare archival materials, as well as faculty articles in PDF format intended for Amherst’s new open access repository. If time permitted, Amherst staff also hoped to explore how Archivematica interacts with the MODS metadata records they created for digitized archival materials. One of the major goals for Amherst College was to assess how Archivematica fit into, expanded, or duplicated current workflows and processes. ● Hampshire College planned to test Archivematica’s capacity to transfer and ingest a selection of text (docx, pdf, html), image (tiff and jpg), audio (wav, mp2), and video (mov, avi, mpg) formats. These formats were chosen because they represent a sampling from across Hampshire’s digital holdings. Hampshire holds a large amount of born-digital audio and video, and was interested to see how well Archivematica worked for those digital objects in particular. ● Mount Holyoke College planned to focus on importing metadata (descriptive and administrative) alongside files. They also wished to explore the end results of the Archivematica workflow, trying to understand the Archival Information Packages (AIPs) and Dissemination Information packages (DIPs) created (Lavoie, 2016), and their potential for migrating objects across content management systems or repository environments, e.g. DSpace, CONTENTdm, ResourceSpace. ● Smith College planned to explore born-digital accessioning workflows and test the usefulness of Archivematica as a tool to assist in preparing born-digital content for ingest. They planned to begin by focusing on newer born-digital accessions received through modern means and media (via email attachments, web sharing services, flash drives, CDs, etc.). In the second phase of the project they hoped to explore migrating content from legacy media for ingest. ● University of Massachusetts Amherst, like Smith College, planned to work on managing born-digital materials, focusing on existing collections of born-digital objects and experimenting with issues of migration from UMass’s current method of Submission Information Package (SIP) storage into an Archivematica environment. These varied approaches were intended to allow for a broad range of digital preservation scenarios and workflows to be assessed and explored simultaneously, which was an express advantage of the consortial model undertaken by the Task Force for the pilot. Each institution reported out on their decisions, workflows, and policies as well as the results of their local Archivematica testing to the group at various check-in meetings over the course of the pilot. A mid-point pivot would shift the focus to the second phase of exploration outlined in the goals for each institution. Creation of the Archivematica Working Group and Project Timeline The installation and configuration of the shared Archivematica platform took place between June and September of 2014 (see ​A Consortial Configuration for Archivematica below). In December 2014, the Archivematica working group was convened. The group consisted of some members of the Five College Digital Preservation Task Force, college archivists, and graduate student interns from the Simmons SLIS West program: Sarah Walden (Project Coordinator, Amherst), Shaun Trujillo (Technical Lead, Mount Holyoke), Aaron Rubinstein (UMass), Leslie Fields (Mount Holyoke), Margy Jessup (Smith), Angelina Altobellis (Hampshire), Christina Barber (Amherst), Wendy Essery (Smith), Johanna Radding (Amherst), and Alexis Dhembe (Amherst). As part of the formation of the working group, the Project Coordinator and Technical Lead determined the shape of the rest of the pilot. They envisioned that the project would unfold in two phases over the next six months: in Phase One (December February) each institution would install and become familiar with the Archivematica virtual machine provided by the Technical Lead by ingesting a few packages of digital files. Once all institutions had a basic familiarity with the software, they would “pivot” to Phase Two (March May). In this phase, each institution would tailor the files that they ingested to specific goals and questions that they wished to answer about Archivematica. Once Phase Two wrapped up, working group members would share lessons learned and outcomes with each other, so that all institutions could benefit from the explorations of each member. Despite the expectations and best efforts of the working group, a significant amount of time during the first phase of the pilot went towards troubleshooting, and ended up cutting into time originally designated for the second “in-depth” phase (see ​Challenges, Pitfalls, Lessons Learned​ below). Nevertheless, the working group stuck to the original timeline, learning a great deal along the way about the challenges of using Archivematica in a consortial model (Figure 1). Figure 1: Archivematica Pilot Timeline, by Johanna Radding A Consortial Configuration for Archivematica After the Task Force assembled a working group to investigate Archivematica as an experimental platform for engaging digital preservation workflows and decision points, the Technical Lead installed several test instances of Archivematica using cloud-based hosting options provided by Amazon Web Services (AWS). Between July and September 2014, the Technical Lead relied on AWS to make a quick and inexpensive assessments of the technical resources necessary to the upcoming pilot. AWS provided the benefit of working outside of the restrictions of any particular campuses’ IT or network infrastructure. The ease with which one is able to configure a web application with AWS also allowed for the Technical Lead to achieve a better understanding of the relationship between Archivematica’s Client and Storage Service, gaining insight not immediately apparent from the platform’s online documentation, by testing several configurations of the software and varying the installation process between tests. The Technical Lead, once able to ascertain specific limitations of the platform by actually installing and testing it, then brought their concerns and recommendations back to the working group for further consideration. One fundamental commitment of the Task Force that greatly impacted the subsequent success of the working group and the overall pilot was its emphasis on consortial collaboration in regard to digital stewardship. As a result the working group made a distinct choice to enforce a shared configuration of Archivematica in which each institution would have a unique client that connected to a centralized communal object store. This way each collaborator could test their own local workflows and standards for describing and accessioning digital content, but the resulting Archival Information Packages (AIPs) would be available for comparison and management in a shared environment. The thinking behind this approach was that a later phase of the pilot would involve reconciling local practices in a shared repository and would fuel broader conversations of a best practice for digital preservation in the Five Colleges. In other words, the working group wanted to ensure that each collaborator would be on a similar, if not the same, page while testing Archivematica and that later on the group would be better situated for comparing and contrasting the results of the pilot. This emphasis on shared storage and a common environment for more transparent and controlled testing of the platform would quickly become an area of technical challenge during the planning stage of the pilot. The most immediate obstacle for implementing a shared installation of Archivematica was setting up secure network communication between each client machine at each institution and a centralized storage server. Archivematica requires two-way communication between the Client application, installed on a local machine, and the Storage Service application, installed either locally or, as in the case of the pilot, on a remote server. Additionally, the requirements of properly installing and configuring each individual Archivematica Client loomed large. Finally, the shared repository approach required finding a suitable place to install and host the centralized storage server. After communicating their concerns with the working group, the Technical Lead consulted with networking staff at the Five College Libraries and was directed towards a feasible technical framework for the pilot. To avoid the work and coordination required to install Archivematica on a “bare-metal” Ubuntu Linux machine at each institution, the Technical Lead decided to work with emulated virtual machine environments [4]. The use of virtual machines alleviated the necessity of installing Archivematica at each institution and, by setting up a virtual private network (VPN) on the centralized storage server, the Technical Lead was able to sidestep the bugbear of configuring identity services, such as Shibboleth or LDAP, or routing connections through network firewalls. Each virtual machine was able to seamlessly communicate with the central storage server via passwordless secure shell (SSH) key authentication over the VPN. All that was required of each working group participant to set up their local instance of Archivematica was installing and running a virtual machine monitor software, e.g. VirtualBox, on a local host machine of their choice (Mac, Windows, Linux, etc.). The participant would then receive a “prepackaged” virtual machine with the Archivematica client (version 1.2) fully installed and ready to run. The Technical Lead created five distinct virtual machine images, one for each participating institution, with all of the necessary SSH keys preinstalled and the emulated Archivematica environment configured to securely transfer data to the centralized Storage Service (version 0.4). Since running the Storage Service server required significant networking and storage resources the Technical Lead arranged to have it installed at their home institution, Mount Holyoke College (Figure 2). Figure 2: Archivematica Client VMs & central Storage Service layout, by Shaun Trujillo Challenges, Pitfalls, & Lessons Learned At the start of the testing phase of the project there were some significant performance issues that arose from the experimental approach of implementing a shared storage server and the need to transfer Submission Information Packages (SIPs) over HTTP. Preconfigured upload limits and directory quotas on both client applications and the storage server affected issues ranging from failed transfers to filling up temporary memory with large files during normalization. Testing ingest and transfer with small files was successful, but restricting transfers to one or two files at a time and the inability early on in the pilot to transfer large files inhibited exploration of Archivematica’s full functionality and defeated one of its primary purposes -to automate batch processing of digital objects. It became obvious during initial testing that the centralized storage configuration compromised system performance. Many of the transfer issues would have been resolved had each institution configured their client to point to a local storage system. The single storage destination became a bottleneck for asynchronous work across the group that would reveal bugs and problem areas at different times. The working group also came to the conclusion that the software’s performance would be improved by installing it directly on a dedicated Ubuntu machine, rather than as a guest virtual appliance on a host Windows or Mac machine. Toward the end of the pilot, the Technical Lead provided an alternative virtual appliance to the group with the Client application and Storage Service installed on the local host machine, which is more in line with the prescribed installation of Archivematica. ​Two of the five​ ​institutions chose to install the standalone instance ​and performance was greatly improved when the Client did not need to communicate with a remote Storage Service. This also ​allowed for continued testing of Archivematica functionality, such as the administrative functions and metadata creation, beyond the scope of the pilot. Working through and resolving these unforeseen technical issues made it challenging to meet the original institutional goals within the timeline. A mid-point pivot had been planned to shift the focus to more in-depth testing of Archivematica’s administrative functionality, however resolving the transfer issues limited the time spent on the second phase. In addition to the above issues, project members encountered some challenges with the Archivematica (version 1.2) interface. A particularly troublesome issue was the lack of error messages during failed transfers. The process would get stuck during initial transfer or during the ingest stage, with no explanation, nor did failure reports provide a specific cause for the error. In some instances failure logs were generated on the storage server, but finding these logs and troubleshooting the errors required technical expertise, so it was not easily undertaken by individual members and had to be managed by the Technical Lead. Some members had anticipated using Archivematica to ingest locally digitized content, generating technical and preservation metadata and creating Dissemination Information Packages (DIPs) for access. However, they encountered issues with creating DIPs during initial ingest, which regularly caused a failure in the Archival Information Package (AIP) transfer to storage. This was a known issue in the Archivematica user community at the time, and the solution recommended by the vendor, Artefactual, was to create the DIP and the AIP in two separate steps, a more cumbersome workflow than that which project members had originally envisioned. Since several of the Five Colleges already had robust tools and workflows in place for digitization, including object description, creating access copies, and storing preservation files, it brought into question where exactly Archivematica would fit in. It became clear that, while Archivematica is well-suited for workflows related to born-digital materials, it is less appropriate for digitized content management. Despite some technical challenges, the group benefited from the testing as a learning experience. ​Archivematica requires some knowledge of digital preservation concepts, as there are a number of decision points along the way, including decisions related to generating preservation metadata, performing format migration and normalization, and providing verity checks for files. ​The project helped to clarify potential workflow steps and metadata requirements for managing born-digital content and to identify steps required to prepare digital accessions for preservation and access. Members also gained a greater appreciation for the benefits of collaboration, such as having the technical and moral support of colleagues struggling with the same issues, especially for the lone digital archivists and those with limited technical support at their home institutions. Working as a group also allowed members to divide up testing of various material types and formats, and to test different functionalities of Archivematica. Each institution reported findings back to the group, shared successes and challenges with each other, and thus as a group covered more ground. A Google Group was created for project members to report errors, share solutions, and elaborate on additional information. Most technical issues encountered were discussed and resolved via that forum, which thereafter served as a record of the group’s activities and highlighted the collaborative process of the working group. Other benefits of the collaborative nature of the project included having the opportunity to explore the challenges of a shared infrastructure; sharing digital preservation practices of each institution; and educating the Five College community about digital preservation practices through publicizing the pilot project and reporting out about its progress. Conclusion Despite many of the challenges faced along the way, the Archivematica pilot project was largely successful. While a permanent consortial instance of Archivematica is as of yet unrealized, the pilot led to a deeper understanding of what is needed to implement a robust digital preservation program across the Five Colleges. Some of the goals of this project were for each institution within the consortium to evaluate their own capabilities and readiness for digital stewardship, to gain the necessary skills and knowledge for digital preservation, and to define policies surrounding digital preservation. This project not only successfully accomplished these goals, it also led to many positive outcomes that are ongoing. At the same time Amherst College was participating in this project, they were also conducting a gap analysis of digital preservation. These two projects side by side led to a much more detailed understanding of where they were with digital preservation workflows and practices. This led to conversations around preservation planning, where digital preservation fit into the larger picture within not only the Amherst College Library but also within the larger institution as a whole and a path to move forward with short term, mid term, and long term preservation goals. Amherst College is now in the early phases of implementing digital preservation policies and born digital workflows largely influenced by the knowledge gained from the Archivematica pilot. Mount Holyoke College, as a result of this project, was able to articulate and make clear to the Archives and to the Digital Assets and Preservation Services department what was lacking in their digital accessioning workflow. One development that was inspired by the workflows explored in the Archivematica pilot was a standardized method for accessioning event video created by the Media Services department along with "automatic" metadata generated from patron request forms. This workflow effectively creates SIPs that could then be processed in Archivematica (or a similar system) to create AIPs for long-term stewardship. Sometime in the near future Mount Holyoke would like to implement a production instance of Archivematica and standardize their born-digital accessioning workflows around it. At Smith College, the pilot project became the impetus for developing digital accession workflows and guidelines for a variety of materials, including physical computer media, electronic transfers, and born-digital AV. The project also prompted Special Collections staff to learn about preservation metadata and how best to utilize output from Archivematica for collection management and access. Smith is not yet using Archivematica in full production, but has recently installed Archivematica version 1.5 on a standalone workstation and hopes to move from testing to production within the year. For University of Massachusetts Amherst this project sparked new thinking about where gaps in digital preservation existed and led to discussions on further developing preservation functionality for their digital repositories. UMass Amherst has also investigated the possibility of contracting a DuraSpace hosted Archivematica service for their archival digital content called ArchivesDirect (2016). It should be noted, too, that Archivematica has been identified as a possible affiliated system that should be considered for integration with the developing Three College Digital Library (3CDL) project. Briefly, the 3CDL is an initiative by Hampshire, Smith, and Mount Holyoke to publish digital objects in a shared Islandora platform (hampshire.edu, 2016). The open source community has already put forth effort to integrate Islandora and Archivematica in a framework called Archidora, which the 3CDL team will evaluate in a later phase of the repositories development (DuraSpace, 2016). Finally, for the Five College Consortium and the Five College Digital Preservation Task Force, there is now a shared basis of experience for understanding digital preservation needs and building out future requirements at each of the participating institutions. Working relationships that resulted from this project have helped establish new rounds of conversation and progress around digital preservation practices. One recent example was the Five Colleges coordinated subscription to and implementation of the Internet Archive’s Archive-It web archiving service during the 2015-16 academic year. Practitioners drew on lines of communication which were established during the Archivematica pilot to coordinate their subscription to Archive-It, which resulted in all five institutions receiving a consortial discount from the Internet Archive. Additionally, colleagues across the Five Colleges (many of whom were pilot project working group members) have met informally to share best practices and advice as they implement web archiving at their individual institutions. In the end, though workflows and readiness amongst the institutions in the Five Colleges continue at varying levels of implementation, the project allowed for each institution to build closer alignment in readiness and in the understanding of what is needed to move forward. Long term digital preservation at a consortial level is an area worth exploring and developing further. Digital preservation is a broad endeavor and a rapidly developing facet of digital collections and institutional repositories, yet is often an area that isn’t fully understood or implemented by many libraries and archives, largely because institutions lack the necessary resources to do it alone. The Five Colleges are committed to pursuing the long-term preservation of their digital resources, and each institution is implementing their own workflows and policies toward that goal; nonetheless each institution shares in the fact that their need for a digital preservation infrastructure is growing. It is critical and optimal to work together toward meeting these common needs. By working collectively, sharing resources, knowledge, and support, the Five Colleges can commit to digital preservation and create long term sustainable digital preservation programs. The Archivematica pilot was a crucial step toward that possibility and provided a small-scale, yet practical model for future collaboration. The pilot forced member institutions to make hard decisions and have conversations about what they were and were not doing with regards to digital preservation. By focusing their inquiry around a common preservation tool, the group was able to frame their collaboration and its outcomes around something tangible while benefiting from shared knowledge, resources, and collegial support. Notes 1. Courtney Mumma, December 6, 2013 (1:53 p.m.), comment on Owens, 2016). 2. For more on archival information packages and submission information packages, see: Lavoie (2016). 3. One exception was the Council of Prairie and Pacific University Libraries’ (COPPUL) development of “Archivematica as a Service”. At the time COPPUL had a Wordpress blog that documented their development process working with Artefactual. For more see: http://coppul.ca/archivematica 4. For a full overview of virtualization and virtual machine usage see: Matthew Portnoy, Virtualization Essentials (Sybex, 2012).